Manipulating 3d virtual objects using hand-held controllers

ABSTRACT

Some examples provides a set of frameworks, process and methods aimed at enabling the expression and exploration of free-form and parametric 3D shape designs enabled through natural interactions with a hand-held mobile device acting as a controller for 3D virtual objects. A reference plane in a virtual space generated by the location of the mobile device may be used to select a 3D virtual object intersected by the reference plane. Positioning of the mobile device may also be used to control a pointer in the virtual space. In an example, the orientation of the mobile device may be detected by an accelerometer or gyroscope. In example, the position of the mobile device may be detected by a position sensor.

CROSS REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/294,837 filed Feb. 12, 2016, which is fully incorporated hereinby reference.

BACKGROUND

Currently, product design and manufacturing are the purview ofenterprises and professionals such as engineers and artists. Everyonehas ideas but only a select few can bring them to reality. The directmanipulation of content on touch-enabled devices is inherentlytwo-dimensional (2D). The design of 3D objects is still predominantlyimplemented through 2D WIMP (Windows-Icons-Menus Pointers) based designmetaphors that require extensive training, and inhibit the ability ofusers to create, manipulate, and modify virtual shapes in astraightforward manner. This is a tremendous waste of untapped humancreative resources and economic potential to the society and worldeconomy.

BRIEF DESCRIPTION

Some examples describes a set of frameworks, process and methods aimedat enabling the expression, manipulation, and exploration of 3Dfree-form shape designs enabled through interactions between a humanpractitioner and a mobile device (such as a smartphone or a tabletcomputer) that is held by the practitioner. In an example embodiment, wedemonstrate how users can directly create, modify, and compose 3D sweptsurfaces through tilt and touch interactions on the mobile phone. Inanother example, clay like objects are created and manipulated like theywould be expected to behave by humans. Finally in another embodiment,all these individual shape interactions are brought together in anenvironment, where multiple objects with different behaviors co-exist inone environment. In all these interactions of the embodiments, theshape, the mobile controller, and the contexts interact creating avirtual environment and objects in it that behave intelligently.

This summary is provided to introduce the selection of concepts in aform that is easy to understand the detailed embodiments of thedescriptions. The embodiments are then brought together in a finalembodiment which described an environment, thereby stressing that eachof the embodiments may be viewed in isolation, but also the synergiesamong them are very significant. This summary is not intended toidentify key subject matter or key features or essential featuresthereof.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The above and other objects, features, and advantages of variousexamples will become more apparent when taken in conjunction with thefollowing description and drawings wherein identical reference numeralshave been used, where possible, to designate identical features that arecommon to the figures, and wherein:

FIG. 1 illustrates the integration of mobile interaction metaphors usingtilt-touch techniques for intuitive shape conceptualization.

FIG. 2 illustrates a Web-based system architecture of some examples.

FIG. 3 shows a Bluetooth-based architecture for mobile devicemodification of 3D image.

FIG. 4 is an illustration of the floating canvas and laser pointermappings.

FIG. 5 illustrates 3D selection and view manipulation using the FloatingCanvas and Laser Pointer mappings.

FIG. 6 illustrates region selection using the Floating Canvas and LaserPointer mappings.

FIG. 7 illustrates view manipulation using the Floating Canvas and LaserPointer mappings.

FIG. 8 illustrates the Floating Canvas interaction for the creation ofplanar curves in 3D space.

FIG. 9 illustrates the Floating Canvas interaction for the creation offully 3D curves in 3D space.

FIG. 10 illustrates creation and modification of parametric shapes.

FIG. 11 illustrates search and assembly of parametric shapes.

FIG. 12 illustrates a visual display of the virtual environment and asmartphone that acts as a reference plane in the virtual environment.

FIG. 13 illustrates a smartphone coordinate system and orientationconvention (left) and its relation to the global reference directiongiven by the magnetic north (right).

FIG. 14 illustrates initial alignment of mobile device to the screenduring calibration.

FIG. 15 illustrates six gestures (row 1) that enable the interactionsacross three modeling states: Configure (row 2), Author (row 3), andManipulate (row 4). (0F, 1F, 2F, and 3F denote 0, 1, 2, and 3 fingergestures respectively).

FIG. 16 illustrates a technique for shape creation (a), in which theuser selects the Add Shape menu item and creates a sweep surface usingone or three finger offsetting gesture. For manipulating a shape (b),the user first hovers on a desired sweep surface and selects the shapeusing the double-tap gesture. In the shape modification example (c) theuser modifies the initial section by sketching, creates a sweep surface,and modifies the final section by sketching, scaling, and panning.

FIG. 17 illustrates algorithms for (a) sweep generation and (b) sectionsketching.

FIG. 18 illustrates a menu (left) showing operations that a user canperform in some examples. A color-picker is shown on the right.

FIG. 19 illustrates a user interacting with a system according to someexamples.

FIG. 20 illustrates a controller's coordinate axes (left) that aremapped to that of a virtual reference plane and the position of theplane is computed by radially offsetting it along â_(z) by a distance D(middle). Here, G is a global coordinate system. The projection of thisreference plane on a given 3D model centered at the origin of G definesa moving region-of-interest (mROI) on the model (right).

FIG. 21 illustrates interactions for explicit rotation and scaling. Onefinger swiping on the phone (left) allows users to rotate a 3D modelabout the axis defined by the y-axis of the phone. Users can also scalethe mROI using a two-finger pinching action.

FIG. 22 illustrates a technique for using one finger (top row) and twofinger interactions (bottom row). Users can perform modeling operationssuch as carving, creasing, painting, texturing, and deformationoperations such as pulling, pushing, or twisting.

FIG. 23 illustrates a user interacting with a MobiSpace setup accordingto some examples.

FIG. 24 illustrates (a) a schematic of a typical physical setup used inthe MobiSpace workflow, (b-c) three stages of the MobiSpace workflow:(b) 3D scanning of a physical objects to create corresponding 3D modelswith texture mapped on to the models, (c) interactive modification ofthe scanned shapes using a smartphone in mid-air, (d) spatial assemblyof different shapes during artwork construction, and (e) final artworkor design constructed from the workflow.

FIG. 25 illustrates unit operations achieved by combining mid-airtracking of a mobile device with concurrent multi-touch gestures in theMobiSpace embodiment.

FIG. 26 illustrates a 3D modeling scene displayed on a computer monitorusing Mobi Space.

FIG. 27 illustrates planar shape assembly according to techniques ofsome examples.

FIG. 28 illustrates using MobiSpace to perform 3D modeling scenedisplayed on a computer monitor.

FIG. 29 illustrates two modes of shape modification using the MobiSpacesetup: (a) coarse mid-air inputs to scale a component's size within anassembly model, (b) user capturing the critical geometric features of acomponent on the device's GUI for their subsequent modification usingcontrolled inputs.

FIG. 30 illustrates a high-level diagram showing the components of adata-processing system.

DETAILED DESCRIPTION

The term “drawings” used herein refers to drawings attached herewith andto sketches, drawings, illustrations, photographs, or other visualrepresentations found in this disclosure. The terms “I,” “we,” “our” andthe like throughout this disclosure do not refer to any specificindividual or group of individuals.

Existing computer-aided-design (CAD) tools were developed in thecomputer as a tool paradigm, where it serves as a passive vessel fordesign. These methods failed to fully leverage the computer-as-a-partnerapproach where the digital medium is treated as an active participant ina creative design process. This leaves great room for shape modelinginteractions to explore the possibilities, true potential, and impact ofusing mobile devices towards shape modeling and designconceptualization.

The integration of a variety of sensors into smartphones provides newaffordances for designing interactive processes to use our knowledge inand of the world towards creative on-the-fly shape modeling processes.Various aspects herein refer to example processes, solely for clarity ofillustration, as “MobiSpace,” “MobiSpray,” “MobiCAD,” “MobiSweep,”“MobiForm,” “MobiSpace,” or “MobiSpin.” Specifically-named exampleconfigurations are not limiting, and portions or the entireties of anynumber of the named techniques can be combined together or with otherexamples described herein.

The ubiquity of mobile devices has led to a disruptive change in the wayusers create, manipulate, and share digital information. Methods forusing mobile phones as mid-air input devices in virtual environmentshave been recently proposed. Current research on mid-air mobileinteractions has mainly focused on 3D object manipulation (rotation andtranslation), virtual scene navigation, and scientific visualization.The unique advantage of using smartphones is that they can be used asportable hand-held controllers that extend our mind and body, to create,manipulate, and modify virtual 3D objects on larger screens, personalcomputers, augmented and virtual reality. Mobile technologies provideadditional new affordances for rotational and touch interactions on thesurface, and when integrated with other external sensors or devices canprovide affordances for additional new smart shape modeling interactionparadigms. Using the mobile devices we now design new interactionmetaphors together with processes and supporting algorithms to simplifyonce complex WIMP-based CAD operations. We envision simple spatialinteractions for easy direct shape modeling operations by transformingour bodily motions and rotations of the phone and contextual sketches onthe devices to desired configurations and geometries.

Instead of merely extending our human capabilities with the digitalplatform, some examples provide ways to augment them, thereby reducingthe need for formalized workflows in traditional CAD that inhibitcreation and require expert-level knowledge (which requires specialtraining to acquire).

Some examples are related to building a framework of embodiedinteractions for creation, manipulation, modification, and behavior ofthree-dimensional shapes. Some examples enable both quick as well asdetailed construction of shapes for application domains ranging such asengineering design, product design, industrial design, and digitalthree-dimensional art.

Mobile spatial user interfaces (M-SUI's) have received significantresearch focus towards 3D manipulation, navigation and design onlarge-screen public displays, immersive environments, and mixed-realitysetups. Mobile devices offer a unique combination of computationalpower, wireless data communication, 3D sensing capabilities, ergonomicmanipulability, and multi-touch input mechanisms that can beinstrumental in enabling embodied interactions for shapeconceptualization and exploring new design work-flows.

FIG. 1 illustrates an example of using a smartphone 100 can be used tocreate a shape of a kettle 102 involving embodied interactions with thephone 100 to create a revolved feature 104, a variable section sweep106, reuse through image-based modeling 108, and assembly 110 allthrough direct interactions using metaphorical canvas and a laserpointer. The perceived affordances of such design metaphors cantransform the process of externalization of creative ideas throughphysically relatable human actions. Throughout this disclosure,references to creating objects can refer to creating shapes of thoseobjects or data representing such shapes, unless otherwise indicated.Other handheld or portable computing devices can be used instead of orin addition to smartphone 100.

Instrumented controllers such as gloves, hand-held trackers, and hapticsdevices, have been extensively studied in the context of mid-air art anddesign. Others have developed “Shape Tape” a specialized highdegree-of-freedom controller for 3D curve input. Recently, commercialgaming controllers and customized hardware have also been demonstratedfor 3D modeling. While commercial controllers offer accessibility,tailoring them towards modeling applications requires hardware design toadd to their generic hardware features such as position, orientation,and buttons for expressing mouse clicks. Using a mobile phone 100 as acontroller takes advantage of off-the-shelf accessibility and thepossibility of a richer interface on the controller itself

Existing works have shown the use of a tablet or a phone as a hand-heldcontroller that helps the user select and manipulate (translate, rotate,scale) a 3D object displayed either on a large screen or a head-mounteddisplay or HMD. Using the tilt of the mobile device has been mostcommonly used for 3D interactions. The multi-touch capability of mobiledevices provides additional capabilities for both direct and indirectmanipulations of the virtual objects. To this end, combination of touchand tilt interactions have been utilized for precise object selection,scene navigation, and immersive interactions. A study identified thebest tilt-touch combinations in terms of user performance, motorcoordination, and user preferences. Similar interactions have also beenutilized within 3D exploratory applications such as volumetric dataannotation and scientific visualization. Some examples herein permitperforming shape creation or deformation tasks that, unlike rigidtransformations, are location and shape specific. The ability toapproach, reach, and deform an object varies as the intended location ofmodification changes. Further, the spatiality of interactions also leadsto physical fatigue that is common in mid-air interaction approaches.Thus, instead of considering a 6 DoF spatial interface, some examplesprovide techniques to mitigate interaction fatigue by augmenting a 3 DoForientation control with multi-touch deformation. Some examples containnovel interactions for 3D shape composition, i.e. activities thatinvolve shape creation tasks (e.g. cross-section sweeping) andmodification tasks (bending, twisting) in addition to manipulation.

Interactive design and shape modeling has also been studied with mobileinterfaces. Several works have explored using multi-touch gestures onmobile devices to drive 3D modeling interactions. Multi-touch gesturescontact multiple fingers simultaneously with a touch screen. In amulti-touch drag gesture, for example, multiple fingers are moved acrossthe surface of a touch screen. Others have also proposed using suchdevices for multi-view 3D shape reconstruction. Xin et al. demonstratedthe use of a tablet as an augmented reality (AR) canvas for 3Dsketching, akin to creating wire-sculptures. Scheible and Ojala proposed“MobiSpray,” a system for intuitive graffiti on large physicalenvironments using mobile phones as spray-cans. Lakatos et al. and Leighet al. proposed the use of mobile devices as spatially-aware hand-heldcontrollers in conjunction with hand-worn gloves for 3D shape modelingand animation. Their work was more focused on demonstrating generalinteractions for modeling scenarios rather than exploring a concretedesign work-flow for shape composition. Mine et al. described anddiscussed an immersive M-SUI system and demonstrated an immersiveadaptation of the “SketchUp” application. Though their work provides anexcellent set of guidelines for mobile-based modeling, their focus wastowards an immersive system augmented with additional hardware forpositional tracking. Some examples permit performing creative 3Dcomposition through interaction and work-flow design. Some examplesoperate without the use of any additional hardware or vision basedmethod for explicit position tracking.

Touch-enabled object rotation typically uses the trackball metaphor. Dueto undesirable finger occlusions, multi-touch methods have been exploredto facilitate dual-finger 3D interactions. Recent works have explored6DoF control using single-handed, two-finger interactions. Decle andHachet provide suggestive one-handed gestures to be applied outside 3Dmodels boundaries for manipulation. To facilitate more precise inputsand constrained manipulations, 3D widgets have been explored to augmentmulti-touch gestures. The fundamental and implicit assumption intouch-based rotation (and rigid manipulation in general) is that theuser would orient a 3D object to match the physical orientation of theinteractive medium—the touch-surface in this case. This assumption isnot applicable for tasks such as object deformation where “manipulation”is constrained on the surface of a 3D object rather than the wholeobject itself. Further, since a 3D object may include or consist ofarbitrary geometric features with different surface normals at differentlocations, the physical “flatness” of touch-screens becomes an inherentlimitation for deformation, which can be an orientation sensitiveoperation. This shortcoming makes it beneficial for users to rotate theobject so as to match the general orientation of an ROI with theorientation of the touch-screen itself. To overcome this limitation,some examples provide a constrained (3DoF) mid-air method that allowsusers to perform 3D actions in the orientation in which the deformationis intended.

Most of the literature on deformation-based shape editing comes withinthe purview of computer graphics and makes use of standard interactiontechniques either with a mouse or touch-enabled devices. Thus,mainstream clay modeling software is still predominantly based on WIMPinterfaces in that they employ a brush-based interaction metaphor. Here,a user typically applies deformation to a shape using a single pointeras if painting on the surface of a 3D model with a brush. Additionally,the user also specifies the type and amount of deformation throughtraditional graphical user interface elements such as menus and sliders.This is true even for multi-touch interfaces where there is a potentialfor using additional finger input for creating richer and more intuitiveinteractions. Works by Bærentzen et al. and Paczkowski et al. haveexplored two finger interactions for deforming 3D shapes on tablets.However, the use of multiple fingers for deformation has not beenstudied or evaluated in literature in comparison to the conventional onefinger approach. Some examples evaluate how a simple difference of usingone and two fingers affects the users' performance and perception in ashape deformation activity. Informed by this evaluation, some examplesdefine multi-touch interactions for shape deformation using asmartphone.

Illustrative System

FIG. 2 and FIG. 3 show components of an example system: human andenvironment data capture units, data processing units, andvisualization/feedback units. Also shown are a user 200 and anenvironment. The user 200, in this case, can be a designer or a personintending to create and alter 3D shapes and the environment is theusers' spatial surrounding comprising of certain key elements. In outscenario, these elements can be an empty design studio, or a table-topsurface equipped with some visualization capability or a vertical wallsetup with visualization capabilities. Also, any kind of touch-displaycapability is not necessary for this system to work, but may beseamlessly integrated while retaining the other aspects of this system.The human and environment data capture unit refers to a hardware devicewhich is capable of providing data regarding the locations and motionsof one or many users in the environment in a non-intrusive untetheredmanner, i.e. without the user wearing a device on his/her body for thepurposes of capturing data. In context of this work, such a device couldbe a commodity depth camera such as MICROSOFT KINECT, ASUSFIXTION PRO orthe ZCAM. The data processing units refer to one or many computingdevices which can interpret the data acquired by the data capturing unitand convert those interpretations into meaningful actions inapplications. This unit can comprise of the standard personal desktopcomputer, laptop computer 202 or a programmed embedded system comprisingof microcontrollers like the ARDUINO. Finally, thevisualization/feedback unit is responsible for providing a visualfeedback of the shape modeling processes and operations being performedby the user 200. This unit may comprise of a standard visual displaylike a computer screen, a head-mounted display used in virtual reality(VR) systems or augmented reality displays. Visualization may also occurusing a projector-wall or projector-table arrangements.

Mapping Mobile Device to Virtual Interaction Space

One elemental interaction in modeling is spanning the 3D space throughtranslation and rotation. Some examples provide for interactions thatcan allow the user to specify the location and orientation of a 3Dvirtual object (effective as a local frame-of reference). The spatialspanning interactions combine interactions for direct and indirectmanipulation as follows:

Orientation: Here, the phones gyroscope readings (roll, pitch, and yaw)are directly mapped to the orientation of a 3D object. In oneimplementation, quaternions transfer the device orientation to thevirtual object. In order to provide a consistent mapping between theuser's screen and the mobile device, there may also be a calibrationstep such that the user can specify the initial alignment of the devicewith respect to the screen

Translation: This utilizes the multi-touch capabilities of the phone inorder to provide translation inputs. The choice of combining touch inputwith direct orientation gives two distinct interaction metaphors: (a)Floating Canvas and (b) Laser Pointer.

FIGS. 3 and 4 show several examples. The conceptual model of theFloating Canvas 400 is a plane moving in 3D space with a localcoordinate frame attached to its center. The orientation of thiscoordinate frame is directly mapped from that of the mobile device 402.Given a 3D orientation, the user can translate this plane along thethree axes 404 of the local coordinate frame by using two-fingergestures. Using a two-finger sliding or panning gesture, the user cantranslate the canvas along direction of sliding on the device 402. Thisis similar to panning on the plane of the canvas. Similarly, the usercan translate the canvas along the normal of the plane by using thepinch gesture. This relates to the idea of offsetting the plane alongits normal direction. Further, the decision regarding the sign of theoffset (i.e. whether the value of the offset is positive or negative) ismade on the basis of the direction of pinch. If the pinching happens sothat the fingers come closer, the offset is positive and negativeotherwise.

The physical laser pointer inspires the Laser Pointer metaphor 406 wherethe end of the pointer spans 3D space in the spherical coordinatesystem. Here, the coordinate frame is global, i.e., the origin of thepointer is fixed at a specified location in the virtual space and theuser can vary the length of the pointing ray using two-finger gestures(FIG. 4). These two metaphors for spatial spanning form the basis of theframework of some examples.

3D Selection

FIG. 5 shows some examples of object selection. Given a 3D virtual spacecomprised of a collection of multiple 3D objects, the selection of asingle desired object from this collection is a basic operation. Thesecond basic operation that is necessary is the selection of a desiredregion on a given 3D object. The objects can be parametric (such asengineering parts within assemblies) or free-form (such as the objectsused in virtual clay modeling). Such a selection can be implementedusing the Laser Pointer and Floating Canvas metaphors. Additionalinteraction techniques allow adaptive and context-aware viewmanipulation to increase the efficiency of selection.

Selecting whole objects: The most common technique to select objects ina virtual scene is the ray-casting method. Some examples permit mappingthe definition of the ray in terms of the smartphone screen. Forselecting a desired 3D object within a virtual scene comprised ofmultiple objects, a user can effectively use the laser pointer andfloating canvas metaphors as shown in FIG. 5. In the case of the laserpointer 500, the selection is implemented by defining a virtual ray 502from the smartphone 504 contained within the plane of the phone's screen506. Rotating the phone 504 simply rotates the ray 502 in the virtualscene and the ray's intersection with a given object 508 results in theselection of that object 508. Similarly, in case of the floating canvas510, the ray 512 is defined along the normal of the phone's screen 514as shown in FIG. 5.

Selecting regions on objects: In case where the user wishes to select aspecific region on the surface of a 3D object, some examples usecone-casting. This technique has been traditionally used to refine auser's selection of whole objects in a cluttered 3D virtual environment.Some examples permit selecting regions on a single given object in ascene using this technique. As shown in FIG. 6, the ray 600 defined withrespect to the smartphone's screen 602 is replaced with a right-circularcone 604 around the ray 600. The intersection of this cone 604 with thesurface of a 3D object can define a region of interest 606 on theobject. The shape of the cone 604 can be modified either by the user orthrough a menu selection to modify the shape of the intersecting region.

Manipulating Scene View: The common workflows in existing CAD andmodeling software programs typically separate the actual shapecreation/modification task from the object view manipulation. Automaticview control has been proposed recently for drawing on 3D surfaces andfor a pottery interface for context-aware shape rotation. Theinteraction metaphors of some examples permit users to perform automaticview manipulation and object rotation. This can be achieved in twosteps. The first step involves the detection of events that specify whenthe user wishes to start and stop camera or object rotation. This cansimply be achieved by providing the user with a button on the smartphonescreen that, when pressed, will signal the software system to begin theprocess of changing the scene view or alternately rotate a 3D model inthe scene. Once the user's intention to rotate has been established, thesecond step is to implement the mechanism for controlling the rotationof the camera (or alternately the object). FIG. 7 illustrates how such arotation is implemented. In case of the laser pointer 702, the userpoints at an object in the scene and rotates the smartphone 704 aboutthe axis 706 defined by the normal direction of the phone's screen 708.In the second case of the floating canvas 710, the rotation occurs aboutan axis of rotation 712 that lies on the plane of the smartphone'sscreen 714.

Creation of Curves

By using the Floating Canvas metaphor, a variety of geometric objectscan be created by a user. The most fundamental of these objects arecurves in 3D space. There are different interactions for creating twodifferent kinds of curves: planar (curves that lie on a plane defined in3D space) and spatial (i.e. the curves that do not necessarily lie on aplane in 3D space). Unlike traditional techniques in CAD, the content ofsome examples provide a more direct approach for specifying curveparameters. It also provides advantages over existing 6 DOF interactionsin terms of controllability and accessibility. To transform raw curveinput data into precise parametric form, some examples use sketchbeautification methods and constraint solving methods.

2D Curves in 3D Space: In order to draw planar curves, the FloatingCanvas metaphor can be used with one-point touch gestures to directlydefine the curve plane and geometry. In FIG. 8, the user 800 simplyplaces the Floating Canvas 802 at a desired location in virtual space804 and orients the canvas by using the hand-held smartphone 806. Asdescribed previously, the orientation of the smartphone 806 directlydefines the orientation of the plane on which the curve 808 is beingdrawn. Once the user 800 has defined the location and orientation ofthis plane, the user 800 simply can draw a curve 808 on the smartphonescreen 810 which is then directly visualized in the virtual space 804.

3D Curves in 3D Space: As shown in FIG. 9, in order to fully createthree-dimensional curves (i.e. curves that have both bending andtwisting properties), the Floating Canvas can be used to sequentiallyarrange datum planes, such that their central positions and normaldirections define local 3D curve parameters. Note that the canvas plane900 has a coordinate frame 902 whose orientation is directly controlledby the rotation of the smartphone 904. This coordinate frame 902 willserve as the Frenet frame for each point on the 3D curve 906. Here, theuser first specifies the intent to create a 3D curve 906 using a menu onthe smartphone 904 and then use the offsetting operation by pressing andholding on the screen of the smartphone 904 while simultaneouslyrotating the phone 904 (FIG. 9). This simultaneous use of the gesture(press and hold) with phone 904 orientation can permit the user to“sweep” the virtual canvas and consequently create a curve through asequence of the Frenet frames of the curve.

MobiCAD: Design of Parametric Shapes

Datum planes are fundamental entities in engineering design. Theyprovide spatial references for configuring protrusion profiles andtrajectories. Within conventional CAD systems, this is a tediousprocess, involving a number of operations and manual parametricconfiguration of each plane. Using the Floating Canvas metaphor, wedemonstrated how the whole process of engineering design can betransformed to enable embodied interactions. One aspect of the metaphorsis that the user can directly specify datum planes through tilt-touchcombinations on the mobile device.

Protrusions: In contrast to CAD, various examples herein provide acollocated modeling space for constructing both the curves and theprotrusion. This allows users to interactively explore differentprotruded shapes without switching modes. For example, users can createlinear extrusions using pinch/swipe gestures, uniform and variablesection sweeps, and revolved extrusions (FIG. 10). Using this scheme,multiple adjoining protrusions can be easily composed for creating morecomplex 3D models. The protrusions can also be made subtractive toremove material from existing shapes.

Feature Selection: 3D engineering models are defined by geometric (e.g.surface, edge etc.) and modeling (e.g. protrusion, fillet, chamfer etc.)features, whose form can be modified through direct manipulation offeature dimensions. By using mobile-based region selection metaphors,users can select distinct features and dimensions for modification.

Detailing Operations: By combining the interactions for regionselection, menu navigation, and interactive dimensioning, users can alsocreate fine level features such as fillets, rounds, chamfers, and holeson top of protrusions.

Dimensional Modification: This interface combines expressive touch andmotion-based gestures on a mobile device for dimensional modification.Touch-based gestures can include two-finger pinch or twist to controllinear and angular dimensions. Motion-based gestures on the other handinvolve suggestive motion of the mobile device itself. For example, aturning motion of the device can be used to define angular dimensions(FIG. 10). This system also enables keypad-based alphanumerical inputson the GUI display of the device for precise dimensional control.

Mating Relations: Due to parametric interdependencies within 3Dengineering models, modification of one model feature inherentlynecessitates adjustment of others. This is analogous to “brushing andlinking” in information visualization, where a changes in one form ofdata representation gets automatically reflected in others. In someexamples, there are provided mobile interaction metaphors forrepresenting, visualizing, and controlling parametric relations in 3Dmodels by invoking region selection metaphors, touch-based interactions,and menu navigation.

Assembly: There are also methods for mobile-based interactions tosupport simple and intuitive 3D assembly. In existing CAD applications,the assembly workflow is very tedious as it requires users to explicitlydefine geometric and spatial constraints between interconnectedcomponents. We have developed a more direct approach that uses ourmobile-based manipulation metaphors to enable rapid configuration ofassembly components. In FIG. 10, a user may sketch a drawing 1000 ondevice 1002. Users only need to approximately define the configurations,through, for instance, rotating device 1002. Using 3D constraintrecognition (Siemens 3D DCM, DS Spatial CDS), our modeling systemsautomatically infer and apply appropriate relations between componentsbased on proximal, spatial, and semantic similarities (e.g. surfacemates, full or partial insertions, parallelism, perpendicularity,concentricity, tangents etc.). Additionally, by using this dimensioningscheme, users can define numerical relations 1004 like linear/angularoffsets. In contrast to existing mid-air interactions where users needto switch between modalities to provide precise inputs, our methodallows for both spatial control and precise inputs using the samedevice.

MobiSweep: Sweep-Based 3D Shape Compositions

Sweep surface representations are fundamental in computer-aidedgeometric design (CAGD) and provide a simple and powerful means fordefining 3D shapes of arbitrary complexity. Further, sweep surfacesinherently lend themselves to an intuitive association to the process ofsweeping a 2D shape in 3D space. In this context, we inspire our workwith two observations from CAGD and M-SUI. First, traditionalconstruction of sweeps relies heavily on the procedural specification ofdatum planes as spatial references and 2D curves profiles andtrajectories. Within conventional CAGD systems, this is a tediousprocess, involves a number of operations, and requires parametricconfiguration of each plane. Secondly, even though sketch-basedinteractions are common to both geometric modeling and 2D mobileapplications, their utilization in existing M-SUI's has been severelylimited towards mid-air shape creation.

In some examples, by combining the spatial freedom in mid-airinteractions with multi-touch capabilities of smartphones, workflows canbe constructed to enable expressive design exploration. To this end, asshown in FIG. 11, we introduce an interaction metaphor that uses asmartphone 1100 as a hand-held canvas. Our interaction is an extensionof the free plane casting method proposed by Katzakis et al. By adaptingthe canvas metaphor for sweep surface creation, we demonstrate how userscan directly create, modify, and compose 3D design concepts 1102 throughtilt and touch interactions 1104 on the mobile phone 1100. The broadergoals of our work are to (a) identify methods to combine tilt and touchinteractions for 3D shape conceptualization and (b) explore newmetaphors to associate 3D modeling operations to mobile interactions.

The design goal behind MobiSweep is to strike a balance between modelingconstraints, interaction techniques, and system workflow to enabledirect spatial ideation. There are mainly two fundamental aspects thatwe considered while designing MobiSweep: (a) 3D manipulation and (b)sweep surface generation. For 3D manipulation, the critical aspect underconsideration is to minimize fatigue for precise manipulations andminimize the interaction time for coarse manipulations. Instead ofimposing full mid-air movements, we employ touch gestures to allowcontrolled and precise 3D manipulation of virtual objects. In order tominimize learning time, we take advantage of the fact that most usersare already familiar with multi-touch gestures for manipulating objects.Thus, we define a single context-aware interaction metaphor that: (a)uses multi-touch gestures and (b) is shared between several modelingtasks.

In some examples, the separation of degrees-of-freedom (DoF) can beeffective if the interactions for the task (sweeping a section) aresynergistic with the input mode provided by the device (the smartphone).Based on this, we inspire our approach from the free plane castingmethod by combining direct orientation control with indirect gesturebased position control. For instance, in FIG. 12, we introduce aninteraction metaphor—phone as a reference plane 1200—that emulates theaction of sweeping a sketched cross-section that is held in the user'shand 1202. In doing so, we do away with the procedural specification ofplanes as spatial references for drawing 2D curves to define profilesand trajectories, as is predominantly done in conventional CAD systems.The virtual environment can be depicted on a screen of computing device1204. One advantage of our metaphor, in some examples, is that inaddition to creation, it naturally lends to spatial actions such ason-the-fly bending, gesture-based cross-sectional scaling, and in-situmodification of the cross-sectional shape by sketching.

Mobile Device Calibration

Systems and software on a mobile phone 1300, such as Android SDK,provides the phone orientation in the form of roll 1302, pitch 1304, andazimuth 1306 (FIG. 13). Because the azimuth 1306 of the mobile phone1300 points to the magnetic north, its value is location dependent.Thus, in some examples, the rotation of the phone 1300 with respect toits z-axis (i.e. plane normal) can be initialized to match with that ofthe reference plane.

To do this, as shown in FIG. 14, we first conduct a calibration stepwhere the phone (controller) 1400 is held horizontally with its localy-axis 1402 directly pointing at the computer screen 1404. In thevirtual environment 1406 this corresponds to the horizontal alignment ofthe reference plane 1408 with its center at the origin. We average thefirst 30 frames of the azimuth values to compute a datum value. Thisdatum values is subtracted from all subsequent azimuth readings duringsubsequent interactions.

In order to define the interaction work-flow for MobiSweep, we beginwith the definition of our interaction metaphor—phone as a referenceplane. Given a hand-held phone, we can define a reference plane in thevirtual 3D space with a local coordinate frame. Subsequently, theobjective is to allow the user to specify the location and orientationof the reference plane. We define the following gestures to achieve thisobjective.

Rotate (Referencing 1500 in FIG. 15): Here, the orientation (and hencethe local coordinate frame) of the phone is directly mapped to that ofthe reference plane. Thus, simply rotating the phone results in therotation of the reference plane (FIG. 15: column 1, row 2). In order tocompute the amount of rotation, the azimuth-pitch-roll representationcan be utilized. In some examples, we used the azimuth-pitch-rollrepresentation to rotate the reference plane using the phone (FIG. 13).Let a, p and r be the azimuth, pitch, roll angles respectively.Following the calibration of the smartphone, we first compute therotation matrix of the reference plane given by M=A*P*R. Here, Arepresents rotation about y-axis by angle a, P represents rotation aboutx-axis by angle −p, and R represents rotation about negative z-axis byangle r. Applying this matrix to the datum coordinate frame (FIG. 14)gives the rotated coordinate frame of the reference plane.

Pan (Referencing 1502 in FIG. 15): Using the two finger sliding gesture,users can translate the reference plane on the x-y plane of the localcoordinate system (FIG. 15: column 2, row 2). In any orientation of thecanvas, users can perform in-plane translation by using the two fingersliding gesture. The translation applied to the canvas is given bys_(p)(m_(t)−m_(t-1)) where m_(t-1) and m_(t) are the mid-points of thetwo fingers in the previous and current time frames. Here, s_(p) is apredefined constant denoting the sensitivity of panning. This is similarto in-plane panning as in the case of the free plane castinginteraction.

Scale (Referencing 1504 in FIG. 15): Users can also perform in-planescaling by using a two finger pinch gesture. However, scaling is acontext dependent operation that is allowed only when the referenceplane either contains a sweep section (FIG. 15: column 3, row 3) or isattached to a 3D object during a manipulation task (FIG. 15: column 3,row 4). The scaling factor applied to the reference plane is given byd_(t)−d_(t-1) where d_(t-1) and d_(t) are the distances between the twofingers in the previous and current time frames respectively.

Offset (Referencing 1506 and 1508 in FIG. 15): In order to allow theuser to translate the canvas along it's normal, there are two gestures,namely one-finger press (FIG. 15: column 4, row 2) and hold andthree-finger pinch (FIG. 15: column 5, row 2). When a user applies theone-finger press gesture, the canvas automatically starts moving alongits normal with a predefined speed. Users can also offset the canvas byapplying a three-finger pinch-spread gesture. In this case, themagnitude of offset is defined by s_(o)(1.0−A_(t-1)/A_(t)), where s_(o)is a predefined constant denoting the offset sensitivity and A_(t-1) andA_(t) are the magnitudes of the areas of the triangle formed by thethree fingers of the user. The one finger method provides a quick andrelatively less precise method for offsetting. On the other hand, thethree finger gesture requires more effort but allows for a more preciseand bi-directional control of the canvas.

Sketch (Referencing 1510 in FIG. 15): Given a 3D orientation, users cansketch a curve on the canvas using the traditional one finger movement.Similar to scaling, we allow sketching selectively based on the modelingtask the user is performing (for instance when the user wants tore-define the cross-section of a sweep surface).

Modeling States

The gestures defined for manipulating the reference plane form the basisof the workflow of some examples. For any given state in the work-flow,the input gestures (FIG. 15: row 1) remain the same but the referenceplane takes a different meaning according to the context of the states(FIG. 15: rows 2-4) as defined below:

Configure (S1): In this state, shown in FIG. 16, the reference plane1600 is detached from all existing shapes (if any). This empty plane canbe manipulated to a desired location and orientation in 3D space usingthe gestures described above (FIG. 15: row 1). Such as manipulation mayoccur either during the creation of the first shape of a composition orduring in-situ composition where a user is directly creating one shapeon an existing shape. Alternately, users can also move the referenceplane 1600 in order to select an existing shape in the virtualenvironment.

Author (S2): In this state, the reference plane is attached to thetop-most section of a sweep surface. Users can (a) create a sweptsurface by offsetting (FIG. 15: columns 4-5, row 3), (b) bend and twista sweep surface by rotating the phone (FIG. 15: column 1, row 3), (c)pan and scale a section using two-finger gestures (FIG. 15: columns 2-3,row 3), (c) modify a section's shape by sketching on the phone (FIG. 15:column 6, row 3).

Manipulate (S3): This state involves rigid transformation of a sweptsurface for composing through assembly. Here, the reference plane servesas a container for the swept surface through which users can translate,rotate, or scale the surface. Additionally, users can also copy anexisting shape and reuse a transformed version of the copy within thecomposition.

Modeling Work-Flow

In the work-flow of some examples, the configure state (S1) is the basestate from where users can transition to either the authoring state (S2)or the manipulation state (S3). The transitions between these states areenabled using a combination of menu and gestures. The controllerinterface for MobiSweep is a single-screen Android application thatallows for two distinct modes of interactions: (a) multi-touch input forreference plane manipulation, sketching, and state transition and (b)menu navigation for state transitions and general software tasks. Below,we describe the three canonical examples for creation, modification, andmanipulation of swept shapes.

Shape Creation: The creation of a swept surface involves the transitionfrom the configure (S1) to the author state (S2) (1602 FIG. 16(a)). Forthis, the user selects the “Add Shape” button on the menu, thusexpressing the intent to begin the creation of a sweep surface. Once theuser has expressed the intention to add a shape, the visualrepresentation of the reference plane 1600 changes to a default circularsection 1604. The user can now sweep the section by using the one fingerpress-hold (or three-finger pinch-spread) gestures. This corresponds tothe offsetting operation 1606 occurring along the reference planenormal. By continuously re-orienting the phone during the sweepingprocess, users can create curved sweeps 1608. Users can also modify theswept surface as described in the following section. Once the user hascreated a desired shape, the swept surface can be detached from thereference plane using the double-tap gesture 1610 effectively bringingthe user back to the configure state.

Shape Modification: Once the user has created a swept surface, theauthoring state allows users to modify it as long as the user has notdetached the reference plane from the surface. The reference plane isattached to the top-most section of the sweep surface (FIG. 15: column3). Hence, all interactions performed by the user affect the top mostsection only and correspondingly changes the remaining sections of thesweep surface (FIG. 16(b)). For instance, simply re-orienting thesmartphone results in the rotation of the top-most section effectivelyallowing the user to bend and twist the swept surface. Similarly, usingthe two-finger gestures allows for panning and scaling the top-mostsection of the swept surface.

The modification of the shape of the top-most section involves threesteps. The user first selects the “Sketch Section” button on the menu1612 to activate the sketching mode. Once in sketching mode, the usersimply sketches a desired curve on the smartphone. In some examples, theuser can sketch the section in a single stroke. Every time the userfinishes drawing a sketch, the sweep surface is immediately modifiedaccording to the new sketched section. Thus, the user can simply keepover-drawing the sketch in order to explore different varieties ofshapes. Once satisfied with the modified section, the user finalizes themodification using the “Confirm Section” button on the menu. Similar toshape creation, the swept surface can be detached from the referenceplane by using a double tap gesture.

Shape Manipulation: Manipulation of an existing shape involves two steps(FIG. 16(c)): hover (S1) and selection (S3). Translating the center ofthe reference plane inside a swept surface is defined as hovering on thesurface. The user can select an object by first hovering 1614 on theobject followed by a double tap gesture 1616 on the phone. Similarly,using the double tap on a selected object reverts the state to hoveragain. Thus, double tap acts as a toggle between the attachment anddetachment of a shape from the reference plane. The use of double-tapenables users to perform selection without looking at the controller.Selection signifies the attachment of a 3D object with the referenceplane, i.e. all rigid transformations applied on the reference plane aretransferred to the selected object. In addition to manipulation, thehover state can also be used to perform operations such as copying,deleting, and coloring a shape by using the menu.

Algorithms for Geometry Creation and Modification

Sweep Surface Generation The sweep surface is represented as a stack ofcross-sections. Once the users starts the offsetting interaction, thesweep surface is incrementally generated in three steps: (a) adding anew section and (b) translating the top-section along the referenceplane normal at until a stipulated time has elapsed, and (c) repeatingaddition and translation as long as the user is offsetting the referenceplane. This process of incremental generation provides the visualcontinuity of sweeping to the users and the translation time defines thedistance between consecutive sections.

In this work, we implemented a variant of the control-section basedsweeping technique wherein every sweep surface can include or consist oftwo control sections at the two ends of the sweep surface. Each controlsection comprises of equal number of points and the information aboutits local coordinate frame (i.e. the frame of the reference plane).Hence, there is a one-to-one point correspondence between the controlsections. For a given pair of control sections, we interpolate eachmeridian of the sweep surface by using the cubic Hermite basis functions(FIG. 17(a)). The interpolation uses four boundary conditions, namely,the position and tangents 1700 at the end points 1702. These areconveniently provided by the vertices and the normal of the section's(1704, 1706) local coordinate frame respectively. Our approach removesthe need for explicit computation of the individual sectiontransformations and avoids frame rotation minimization and sectionblending. This simplifies the operations (e.g., bending, twisting,panning, scaling, or section modification) in the authoring state.

Section Modification: Some examples permit single stroke sketching andthe number of points in each section of the sweep surface is constantand pre-defined. Illustrated in FIG. 17, for a sketch input 1708, wefirst determine if the sketch is an open or a closed curve based on asimple distance threshold between the two end-points of the sketchinput. For a closed curve, we implemented a three stage processing ofthe sketch input (FIG. 17(b)). First, we perform an equidistant curvere-sampling 1710 to match the number of points on the sketch to theinitial control section of the sweep surface. Subsequently, we determineif the orientation of the curve is the same as that of the initialcontrol section. This involves the comparison between the signs of theareas enclosed by the sketched curve and the initial section. If theinitial and sketched sections have opposite orientations, we correct thesketch orientation by reversing the order of vertices in the re-sampledsketch input 1712. Finally, we minimize the twist between the sketchinput and the initial section.

Menu Implementation

The smartphone application menu (FIG. 18) allows users to start thecreation of a sweep surface (Add Shape), provide sketch inputs formodifying sweep sections (Sketch Section), reusing shapes using the CopyShape operation and perform general tasks (Delete Shape, Save All, andSave & Clear). The transition from the configure (S1) to the authorstate (S2) is achieved by selecting the “Add Shape” button on the menu.This signifies the intent to begin the creation of a sweep surface. Oncethe user has expressed the intention to add a shape, the visualrepresentation of the reference plane changes to a circular section. Theuser can now sweep the section, bend or twist a swept shape, pan andscale a section, or sketch the shape of the section. In all these cases,the reference plane remains attached to the most recent section of theswept surface. Once the user has created a desired shape, the sweptsurface can be detached from the reference plane by using a double tapgesture, hence bringing the user back to the configure state (S1). Inorder to apply a color to a shape in a given 3D composition, the userfirst hovers on a desired shape in the composition. Subsequently, asshown in FIG. 18, the user uses the “Pick Color” button 1800 on the menuin order to specify the color for the shape using the color pickerdialog 1802 on the smartphone app.

Mobiform

Various examples describe “ubiClay,” an embodied approach for digitalclay-like 3D modeling. Our main idea is to re-purpose a smartphone as ahand-held proxy for virtual clay, resulting in multi-touch asymmetricbi-manual shape deformation. Guided by the evaluation of touch-basedshape deformation, we designed an interaction metaphor that represents asmartphone as a moving region of interest (mROI) on the surface of avirtual 3D model displayed on a larger screen. The direct control of thephone's orientation allows the user to navigate to a region on thesurface of a 3D model and simultaneously deform the model usingmulti-touch gestures on the phone's screen. The combination of directorientation control with multi-touch input partly emulates theperception of deforming a real piece of clay held in the user's hands.

System Setup

Shown in FIG. 19, the ubiClay interface comprises of a hand-heldcontroller (smartphone) 1900, and the virtual environment 1902 (i.e. amodeling application running on a personal computer). We used a 12″Microsoft Surface Pro3 tablet 1904 as the personal computer, and theSamsung Galaxy Note 3 as the hand-held controller that served as thetangible proxy for clay. We implemented a one-way Bluetooth 304 serialport communication to stream input data from the controller (phone) 1900to the ubiClay application (running on the tablet) 1904. The input datapacket included device orientation and touch coordinates of fingers. Ourcontroller interface was implemented using the Android SDK and theapplication was developed in C++ with OpenGL Shading Language forrendering.

Postural ergonomics was also an important factor in our setup. In orderto provide a comfortable posture to users 1906, the virtual environment1902 was placed horizontally at a height lower than that of the user's1906 sitting height (FIG. 19). This allowed the user 1906 to manipulatethe controller in a constrained mid-air fashion while taking armsupport.

Interactions

Phone as a Moving Region-of-interest: In FIG. 20, we begin with the mostelemental interaction of ubiClay: selecting a region of interest (ROI)on the surface 2000 of the 3D model 2002. Given a hand-held phone 2004,the objective is to allow the user to specify the ROI on the surface bydirectly controlling orientation of the phone 2004. As a first steptowards this objective, we define a virtual reference plane P 2006 witha coordinate frame, F=(A, O), whose orientation is directly mapped tothe coordinate frame of the smartphone 2004 (FIG. 20(a)). Here,A=(â_(x), â_(y), â_(z)) and O=(o_(x), o_(y), o_(z)) are the axes andcenter of the coordinate frame respectively. Subsequently, we translatethe center O 2008 along â_(z) by some arbitrary distance D 2010 (FIG.20(b)). Finally, the virtual reference plane 2006 is orthogonallyprojected on the 3D model 2002 along the ray {circumflex over(r)}=−â_(z) (FIG. 20(c)). Any point p in the plane P can be similarlymapped to a point q on the 3D model 2002. In some examples, this is aray-casting approach and converts the phone as moving region-of-interest(mROI) on the surface of a given 3D model.

There are two modes of interaction in ubiClay: surface navigation andsurface modeling. In this work, our aim is not to build a specificsystem. Thus, instead of mapping each user action with a specificmodeling or navigation operation, we identified a canonical set ofinteractions that can be used intuitively for a broad category ofmodeling operations. These modeling operations could be rigidtransformations (such as rotation and scaling), sculpting (such aspulling, pushing, or carving) or detailing operations (such as drawing,painting, or texturing). Recent work characterized the expression ofdeformation intent based on hand grasp and motion in mid-airinteractions. Based on this, for each of our interaction modes, weorganized our interactions on the basis of the number of fingers andtype of motion.

Surface Navigation 1-Finger: The mROI interaction allows users to onlynavigate on the front-facing region of a 3D modeling. Thus, as shown inFIG. 21, we also provide a one finger method for rotating the object2100 similar to the tablet interfaces described in the previoussections. Here, the orientation of the y-axis 2102 of the phone 2104(i.e. â_(y)) defines the axis of rotation 2106 and a one-finger swipingaction 2108 of the phone 2104 defines the angle (FIG. 21).

2-Finger: Also in FIG. 21, a typical operation in navigation is scalingor zooming that allows users to modify the object 2100 at differentresolutions. In ubiClay, this corresponds to scaling the mROI on theobject's surface and is performed using the two-finger pinch/spreadgestures 2110.

Surface Modeling 1-Finger: In the modeling mode, one-finger actions canbe intuitively mapped to inward deformation operations such as pushingand creasing and drawing operations such as painting and texturing (FIG.22, top row). Here, the trajectory of the finger on the phone is mappedto the mROI, and so permits transferring a curve from the phone to the3D model.

1-Finger Hover: One of the most important issues in ubiClay interactionsis potential splitting of visual attention while performing preciseone-finger operations (especially drawing curves on a 3D model). Duringa drawing operation, the user can either look at the physical surfacethat is being drawn upon (i.e. the phone 2200) or at the virtual model2202 on which the curve 2204 is being mapped. Thus, some examplesovercome the lack of a visual feedback as to where the user ispotentially going to start drawing in prior schemes. For this, weintroduce a one-finger hover interaction. Here, the location of theuser's finger in proximity to the phone 2200 is mapped and displayed onthe 3D model 2202 without activating the drawing operations.

2-Finger: We map two-finger pinch/spread action to the pulling/pushingdeformation operations (FIG. 22, bottom row). The deformation isachieved by performing a pinching gesture on the object 2206. Thedeformation handle 2208 is defined by the mid-point of the two touchpoints and the displacement is defined by the amount of pinch asfollows:

δ_(h) =−k(d _(t) −d _(t) − ₁)  (1)

Here, k>0 is a constant and d_(t) is the distance between the twofingers (in the pixel space) at time t (measured in frames). Note thatwhile a pinch gesture would allow for an outward displacement (akin topulling), spreading the two fingers allows for inward displacement ofthe mesh. We determined the value of k=0.001 through pilot experimentsso as to keep the one and two finger approaches similar in terms oftheir perceived responsiveness (i.e. speed of deformation). In additionto pushing and pulling, we also propose a twisting operation with twofingers. In this case, the action performed by the user is similar tothe established 2D rotation gesture in touch-based interactions.

3-Finger: Operations such as smoothing can be used in digital claymodeling according to some examples. In ubiClay, this can be achieved athree-finger approach for smoothing of the 3D model surface. This actioninvolves swiping on the phone's screen with three contact points.

In some examples, in order to permit more controllable interactions, themROI is constrained to lock to its orientation upon detecting a touch ora hover. This can reduce jitter due to the manual control of the mROI,particularly during simultaneous action of orienting the phone andperforming touch interactions.

Mode switching Note that both one and two finger interactions can bemapped to either navigation or modeling modes. Thus, we needed anexplicit method for the user to switch between these modes. Afterexperimenting with different options (menu on the phone, menu on thevirtual environment, widgets on the phone etc.), we found that theproblem of split attention adversely affected the overall interactionworkflow. We used the physical volume button, located on the side of thephone, to allow users to switch between navigation and modeling modes.We believe that this approach (if comfortable for users) is scalable toadditional modes in a design workflow and also adds to the tangibilityof our proposed interactions.

Mobispace: Mid-Air 3D Modeling Using Tracked Mobile Devices

Mid-air inputs allow users to interact with virtual 3D interfaces usingsuggestive actions in physical space. Compared to traditionalmechanisms, where 2D inputs with a mouse and keyboard are mapped into 3Doperations, mid-air inputs provide a more direct and efficient means tomanipulate and configure virtual 3D elements. The motion gestures usedin such interactions are also easy to learn and apply due to theirsimilarity to actions used in everyday human experiences (e.g. picking,placing, manipulating, pulling, and bending). They also provide a senseof perceptually consistent correspondence between what the user is doingin physical space and the expected outcome in the virtual space, thusbridging the gap between user inputs and system behavior.

Mid-air interactions have been primarily explored as freehand gesturesand digital controllers. While free-hand gestures provide flexibility intheir usage, they also suffer from issues such as sensitivity to handocclusions, low tracking fidelity, and inability to provide tactilefeedback. Digital controllers on the other hand use a tangible devicewith powerful electromechanical sensors for 3D inputs. However, suchcontrollers provide limited interactive capabilities as they can beunwieldy to hold and lack adequate event triggering mechanisms. Wepresent a novel method for mid-air interactions, where an ordinarysmartphone is repurposed as a 3D interactive medium. In contrast todigital controllers, smartphones provide several advantages. First,given their growing popularity, they are significantly more ubiquitousand accessible than digital controllers. Second, smartphones provide awide variety of interactive capabilities that are not found in digitalcontrollers. For example, their GUI interface along with multi-touchscreen can be used for close range interactions and to provide preciseinputs. The presence of motion sensors in smartphones also allows forreliable motion gesture tracking, enabling their use as a 3D inputmedium. Users can also provide a variety of expressive event triggeringmechanisms on the multi-touch surface such as tapping, swiping,pinching, twisting, dragging etc.

In some examples, we use a smartphone as a mid-air input device bycombining its innate self-orientation tracking capabilities with spatialposition tracking using an external depth camera. Compared to existingmethods, some examples do not require installation of digital addendumson the smartphone. Some examples use a non-invasive (wireless) methodfor both tracking the 6 degrees of freedom motion of the phone andcommunicating the tracked data to a central computer system. In someexamples of a 3D modeling context, the smartphone serves as aninteractive medium for conducting all 3D operations using mid-airinputs.

3D Tracking and Mapping

In FIG. 23, to track the smartphone's 2300 3D position, we appliedLucas-Kanade Optical Flow (OpenCV library: 2 pyramid levels, 5×5 searchwindow) on the depth data stream (represented as a gray scale images)from the interaction space 2302. We also use pre-defined depth anddwell-time thresholds of the smartphone 2300 with respect to the depthsensor 2304 to detect tracking initialization (50-60 cm, >1.5 sec) anddisengagement (85-100 cm, >1.5 sec). During tracking initialization, thesystem seeks all pixels with depth values within initializationthresholds, and assigns the median pixel as the initial tracking point.When the user places the phone 2300 on the desk, the tracked point onthe smartphone 2300 is maintained within the disengagement threshold,indicating termination of position tracking.

The smartphone's orientation angles are read by the ANDROID app andcommunicated to the PC via BLUETOOTH. Before system usage, we calibratethe phone's twist offset by placing it on the desk (roughly aligned withthe sensor's Y) and measuring the angle between sensor Y and magneticnorth. Subtracting this offset value from the measured yaw angle at eachframe, gives the phone's twist angle about global Z.

We define the depth sensor's 2304 coordinate frame (FIG. 23) as theglobal frame of reference. Since the depth sensor 2304 is facingvertically downwards, the smartphone's position can be linearly mappedinto the 3D scene 2306. In some examples, a 5 mm motion of the phone2300 gets mapped as a 0.12 unit displacement of the planar cursor.Similarly, the phone's tilt (pitch and roll) and twist angles can alsobe directly used as the cursor's orientation in the 3D scene.

Shown in FIG. 24, we implemented a preliminary 3D interface (FIG. 24(a))to demonstrate the MobiSpace workflow. Some examples use off-the-shelfcomponents to reduce the effort of custom implementation and improvesystem robustness. The interface can be integrated within an ordinarycomputer desk 2400 and comprises of: (i) a monitor 2402 to displaydigital content, (ii) an overhead RGB-D sensor 2404 for motion trackingand 3D scanning, and (iii) a smartphone 2406 for interacting with the 3Dmodeling scene.

FIG. 25 illustrates user interactions with this interface during thethree stages of the MobiSpace workflow.

The following sections provide a 3D modeling scenario, where virtual 3Dmodels (acquired from a 3D digital scanning process) are composed into3D collages to support artistic and design ideation. The modeling stepscan be broadly classified as the following activities.

Modify and Compose

We used mid-air interactions for the modify and compose states, as theyallow users to express 3D modeling operations via intuitive actions(e.g. picking, placing, manipulating, pulling, and bending).Additionally, when constructing artwork in MobiSpace, they also providea sense of physical engagement and spatial mobility, relatable to thephysical act of assembling objects.

To select an appropriate mid-air modality, we first explored unimanualfree-hand gestures using Softkinetic and LeapMotion systems. Our testshowever revealed that they either lacked adequate tracking fidelity orhad a small interaction space. We also found bimanual free-hand gesturesunsuitable for subtle shape manipulations required in MobiSpace. Givensuch limitations, we explored using a planar cardboard proxy as asurrogate for the dominant hand. Even though it improved tracking andprovided passive haptic feedback, the proxy lacked a medium forsecondary inputs (e.g. selection, scaling etc.). Thus, we foundsmartphones to be a natural extension of the planar proxy, as theyretain the planar interaction metaphor while providing a collocatedsurface for multi-touch inputs. We acknowledge that hand-heldcontrollers can also provide similar capabilities, but rejected them dueto commonality of smartphones. During mid-air interactions, we use theRGB-D sensor and the smartphone's IMU sensor to track the phone.

In FIG. 26, the 3D composition scene 2600 resembles the interactionspace, and can include or consist of a horizontal desk 2602 over whichscanned shapes 2604 are laid out. To avoid clutter, only 6 shapes aredisplayed at a time, but the scroll arrows 2606 allow access tonon-displayed shapes. Since we use a flat screen display, shadows arerendered on the desk surface to assist user depth perception. Thetrash-bin 2608 is used for discarding unwanted shapes. Users canindicate intent to start interacting with the 3D scene by raising thesmartphone 2610 towards the sensor (2404 of FIG. 24). This gestureactivates a planar cursor 2612 inside the 3D space 2600. Since thesmartphone's spatial parameters are mapped onto this cursor, its 6DOFmotion is directly controlled by holding and moving the phone 2610 inmid-air. To disengage the cursor, users can simply place the phone 2610on the desk 2602. In addition, touch gestures on the phone 2610 allowusers to indicate specific intent like grab, release, scale, andreplicate. The following shape composition operations can be performed.

Shape Selection and Manipulation. A bounding box around a shape appearswhenever the cursor is close enough to select it. As shown in FIG.25(c)-top, the planar cursor is first “sliced into” the shape along aconvenient direction. A single tap gesture is then used to attach theshape onto the cursor, enabling its spatial control via manipulation ofthe cursor. A single tap gesture is also used to release the shape at astationary location.

Design Composition. By using the planar cursor, shapes can beindividually picked up and assembled into a 3D collage. At any pointduring this process, users can rotate or translate the assembly tochange its viewpoint. For translation, the cursor is first brought closeto the assembly center. A touch-and-hold gesture along with the cursormotion are used, respectively, to clutch and translate the assembly. Thesame interaction is used for assembly rotation, except here the cursoris place away from the assembly center. Additional, the cursor's motionis used to pivot the assembly about its center via a rotation handle.Our system also allows creation of sub-assemblies that can be combinedlater.

Shape Modification. A given shape in an assembly can be selected formodification by first bringing the cursor close to it and applying adouble tap gesture. Within this state, a two finger pinch gesture(inward or outward) uniformly scales the selected shape. To deform theshape, users can “grab” either one of its axial endpoints with thecursor using a touch-and-hold gesture, and move the cursor in thedirection of shape elongation, compression, or bending. During shapemodification state, open and closed hand icons (instead of the planarcursor) indicate proximity to and clutching of the shape end-point.

Copy. Any shape within the sculpture can be copied multiple times. Theinteraction for this is similar to shape selection, except here we use atwo-finger tap gesture. This operation is particularly useful forcreating patterns of identical shapes.

Undo. The undo command can be invoked by using a three-finger tapgesture. It allows users to revert a 3D design back for up to the lastfive configurations.

Delete. Users can choose to discard any shape from the design by pickingit up and releasing it over the trash-bin.

Advanced 3D Modeling Operations

To demonstrate advanced 3D modeling operations using MobiSpace, wepresent a planar shape assembly application, where planar shapes arespatially configured to create meaningful virtual 3D models. Here, andas shown in FIG. 27, the smartphone 2700 provides multi-levelaffordances. First, users can directly draw 2702 the profile of planarshapes 2704 on the device 2700 itself (FIG. 27(a)). Subsequently, theycan use the MobiSpace system to spatially manipulate and assemble theshapes 2704 one at a time (FIG. 27(b)). Here, because the planar shape2704 is directly drawn on the device 2700, it serves a physicalcontainer for the shape 2704. Given the planarity of the device 2700,users can thus vicariously configure the shape 2704 as if they wereactually holding the shape itself. Using this context, we present thefollowing advanced modeling capabilities that can be achieved usingMobiSpace.

While mid-air inputs with a mobile can provide reasonablecontrollability during 3D shape manipulations, it does not afford theprecision found using GUI-based tools. In our system, we utilizecontext-specific constraints on the mid-air inputs to interpret them asmore structured 3D interactions. Such constraints are automaticallyinferred by the system, and used to assist design of structuredgeometry, complex details, and aesthetic features.

Assembly Constraints. During assembly, if a newly added shape intersectswith pre-existing assembly shapes, its orientation is automaticallyadjusted to be orthogonal to the adjoining neighbors. This ensuresphysical connectivity of the shapes during fabrication and also providesa structured appearance in the assembly. By imposing orthogonalitybetween adjoining planar shapes, the limited accuracy of mid-airinteractions could be compensated for, allowing users to provide preciseinputs, based on the context. The application of assembly constraints isgeneralizable to more generic forms of shapes, where other geometricrelationships such as mating surfaces, concentricity, and insertions candrive automatic constraint application.

Procedural Operations. MobiSpace enables procedural operations, allowingusers to create regular geometry within an assembly for aesthetic designand structural fidelity. Here, we enable two types of proceduraloperations. In the first, users can create a parallel pattern ofidentical shapes along a linear path 2800 (FIG. 28(a)) or the contour ofanother shape 2802 (FIG. 28(b)). For this, a two finger 2804 or threefinger 2806 single tap gesture indicates the type of patterning, andgenerates a copy of the shape hovered over by the cursor. The device'smotion 2810 then defines the translation and placement of the copiedshape along a constrained path. Users can also create a blended patternbetween two non-identical shapes 2812 (FIG. 28(c)) by first placing twoend point shapes (2814, 2816) over a base shape 2818, and indicating thenumber of intermediate shapes 2820 in the blend menu (displayed on thedevice). The intermediary shapes are obtained by interpolatingcorresponding vertices between the two end-point shape profiles. Thisoperation, in addition to utilizing the 3D interactive capabilities ofMobiSpace, also leverages the GUI display of the device to provide menubased alphanumerical inputs.

Multi-modal Shape Modification. The individual components of a 3D designmodel can be modified either in 3D space or the device's GUI, dependingon the precision requirements and the nature of modification inputs. Weuse the planar shape assembly context to demonstrate such bimodal shapemodification scheme. In FIG. 29, by applying a two-finger pinch gesture2900 (similar to zooming in mobile devices) while hovering the cursorover a planar shape 2902, the shape 2904 is uniformly scaled within theassembly (FIG. 29(a)). This interaction represents modification inmid-air, where a coarse multi-touch gestures is used to indicatemodification operations on a 3D component.

Likewise, in FIG. 29, a 2D geometry can also be modified on the GUIdisplay of the device 2906 29(b)). Here, the critical geometricattributes 2906 are captured on the device 2908, allowing users toprovide controlled and more precise inputs towards its modification2910. Such changes are then reflected in the corresponding shapes in the3D assembly model 2912. This is analogous to brushing-and-linking indata visualization, where identical data within different spacesmaintain a consistent representation.

Throughout this description, some aspects are described in terms thatwould ordinarily be implemented as software programs. Those skilled inthe art will readily recognize that the equivalent of such software canalso be constructed in hardware, firmware, or micro-code. Becausedata-manipulation algorithms and systems are well known, the presentdescription is directed in particular to algorithms and systems formingpart of, or cooperating more directly with, systems and methodsdescribed herein. Other aspects of such algorithms and systems, andhardware or software for producing and otherwise processing signals ordata involved therewith, not specifically shown or described herein, areselected from such systems, algorithms, components, and elements knownin the art. Given the systems and methods as described herein, softwarenot specifically shown, suggested, or described herein that is usefulfor implementation of any aspect is conventional and within the ordinaryskill in such arts.

FIG. 30 is a high-level diagram showing the components of an exemplarydata-processing system 3000 for analyzing data and performing otheranalyses described herein, and related components. The system 3000includes a processor 3086, a peripheral system 3020, a user interfacesystem 3030, and a data storage system 3040. The peripheral system 3020,the user interface system 3030 and the data storage system 3040 arecommunicatively connected to the processor 3086. Processor 3086 can becommunicatively connected to network 3050 (shown in phantom), e.g., theInternet or a leased line, as discussed below. Techniques shown in FIGS.1-29, e.g., for receiving and processing user input, can be carried outby systems or devices incorporating one or more of systems 3086, 3020,3030, 3040, and can each connect to one or more network(s) 3050.Processor 3086, and other processing devices described herein, can eachinclude one or more microprocessors, microcontrollers,field-programmable gate arrays (FPGAs), application-specific integratedcircuits (ASICs), programmable logic devices (PLDs), programmable logicarrays (PLAs), programmable array logic devices (PALs), or digitalsignal processors (DSPs).

Processor 3086 can implement processes of various aspects describedherein. Processor 3086 and related components can, e.g., carry outprocesses for manipulating a 3D image with a mobile device 3004, such asprocesses described above. The mobile device 3004 is manipulated by auser 3038 such as by rotating the mobile device 3004, moving the mobiledevice 3004 through space, touching a touch screen on the mobile device3004, or activating a button or switch on the mobile device 3004. Themobile device 3004 detects these manipulations using onboard sensors andinput devices to provide an indication of the manipulation to theprocessor 3086 or other portion of the data processing system 3002. Theindication may be sent via the network 3050 as a wired or wirelesscommunication. The indication may also be sent via another communicationchannel. The mobile device 3004 can be part of system 3001 or separatetherefrom but communicatively connectable therewith.

Processor 3086 can be or include one or more device(s) for automaticallyoperating on data, e.g., a central processing unit (CPU),microcontroller (MCU), desktop computer, laptop computer, mainframecomputer, personal digital assistant, digital camera such as 3D camerasystem 3006, cellular phone, smartphone, or any other device forprocessing data, managing data, or handling data, whether implementedwith electrical, magnetic, optical, biological components, or otherwise.

The phrase “communicatively connected” includes any type of connection,wired or wireless, for communicating data between devices or processors.These devices or processors can be located in physical proximity or not.For example, subsystems such as peripheral system 3020, user interfacesystem 3030, and data storage system 3040 are shown separately from theprocessor 3086 but can be stored completely or partially within theprocessor 3086.

The peripheral system 3020 can include or be communicatively connectedwith one or more devices configured or otherwise adapted to providedigital content records to the processor 3086 or to take action inresponse to processor 3086. For example, the peripheral system 3020 caninclude digital still cameras, digital video cameras, 3D camera systemssuch as 3D camera system 3006, cellular phones, or other dataprocessors. The processor 3086, upon receipt of digital content recordsfrom a device in the peripheral system 3020, can store such digitalcontent records in the data storage system 3040.

The user interface system 3030 can convey information in eitherdirection, or in both directions, between a user 3038 and the processor3086 or other components of system 3000. The user interface system 3030can include a mouse, a keyboard, another computer (connected, e.g., viaa network or a null-modem cable) such as the controller client, or anydevice or combination of devices from which data is input to theprocessor 3086. The user interface system 3030 also can include adisplay device, a processor-accessible memory, or any device orcombination of devices to which data is output by the processor 3086.The user interface system 3030 and the data storage system 3040 canshare a processor-accessible memory.

In various aspects, processor 3086 includes or is connected tocommunication interface 3015 that is coupled via network link 3016(shown in phantom) to network 3050. For example, communication interface3015 can include an integrated services digital network (ISDN) terminaladapter or a modem to communicate data via a telephone line; a networkinterface to communicate data via a local-area network (LAN), e.g., anEthernet LAN, or wide-area network (WAN); or a radio to communicate datavia a wireless link, e.g., WI-FI or GSM. Communication interface 3015sends and receives electrical, electromagnetic or optical signals thatcarry digital or analog data streams representing various types ofinformation across network link 3016 to network 3050. Network link 3016can be connected to network 3050 via a switch, gateway, hub, router, orother networking device.

In various aspects, system 3000 can communicate, e.g., via network 3050,with a data processing system 3002 or with mobile device 3004, each ofwhich can include the same types of components as system 3000 but is notrequired to be identical thereto. Systems 3000 and 3002, and mobiledevice 3004, can be communicatively connected via the network 3050(e.g., a cellular, BLUETOOTH, or WIFI network). At least one of system3000, system 3002, or mobile device 3004 can execute respective computerprogram instructions to perform aspects of some examples, whetherindependently or in mutual coordination. Some examples include system3000 and mobile device 3004 but not system 3002.

Processor 3086 can send messages and receive data, including programcode, through network 3050, network link 3016 and communicationinterface 3015. For example, a server can store requested code for anapplication program (e.g., a JAVA applet) on a tangible non-volatilecomputer-readable storage medium to which it is connected. The servercan retrieve the code from the medium and transmit it through network3050 to communication interface 3015. The received code can be executedby processor 3086 as it is received, or stored in data storage system3040 for later execution.

Data storage system 3040 can include or be communicatively connectedwith one or more processor-accessible memories configured or otherwiseadapted to store information. The memories can be, e.g., within achassis or as parts of a distributed system. The phrase“processor-accessible memory” is intended to include any data storagedevice to or from which processor 3086 can transfer data (usingappropriate components of peripheral system 3020), whether volatile ornonvolatile; removable or fixed; electronic, magnetic, optical,chemical, mechanical, or otherwise. Exemplary processor-accessiblememories include but are not limited to: registers, floppy disks, harddisks, tapes, bar codes, Compact Discs, DVDs, read-only memories (ROM),erasable programmable read-only memories (EPROM, EEPROM, or Flash), andrandom-access memories (RAMs). One of the processor-accessible memoriesin the data storage system 3040 can be a tangible non-transitorycomputer-readable storage medium, i.e., a non-transitory device orarticle of manufacture that participates in storing instructions thatcan be provided to processor 3086 for execution.

In an example, data storage system 3040 includes code memory 3041, e.g.,a RAM, and disk 3043, e.g., a tangible computer-readable rotationalstorage device or medium such as a hard drive. Computer programinstructions are read into code memory 3041 from disk 3043. Processor3086 then executes one or more sequences of the computer programinstructions loaded into code memory 3041, as a result performingprocess steps described herein. In this way, processor 3086 carries outa computer implemented process. For example, steps of methods describedherein, blocks of the flowchart illustrations or block diagrams herein,and combinations of those, can be implemented by computer programinstructions. Code memory 3041 can also store data, or can store onlycode.

Various aspects described herein may be embodied as systems or methods.Accordingly, various aspects herein may take the form of an entirelyhardware aspect, an entirely software aspect (including firmware,resident software, micro-code, etc.), or an aspect combining softwareand hardware aspects These aspects can all generally be referred toherein as a “service,” “circuit,” “circuitry,” “module,” or “system.”

Furthermore, various aspects herein may be embodied as computer programproducts including computer readable program code (“program code”)stored on a computer readable medium, e.g., a tangible non-transitorycomputer storage medium or a communication medium. A computer storagemedium can include tangible storage units such as volatile memory,nonvolatile memory, or other persistent or auxiliary computer storagemedia, removable and non-removable computer storage media implemented inany method or technology for storage of information such ascomputer-readable instructions, data structures, program modules, orother data. A computer storage medium can be manufactured as isconventional for such articles, e.g., by pressing a CD-ROM orelectronically writing data into a Flash memory. In contrast to computerstorage media, communication media may embody computer-readableinstructions, data structures, program modules, or other data in amodulated data signal, such as a carrier wave or other transmissionmechanism. As defined herein, computer storage media do not includecommunication media. That is, computer storage media do not includecommunications media consisting solely of a modulated data signal, acarrier wave, or a propagated signal, per se.

The program code includes computer program instructions that can beloaded into processor 3086 (and possibly also other processors), andthat, when loaded into processor 3086, cause functions, acts, oroperational steps of various aspects herein to be performed by processor3086 (or other processor). Computer program code for carrying outoperations for various aspects described herein may be written in anycombination of one or more programming language(s), and can be loadedfrom disk 3043 into code memory 3041 for execution. The program code mayexecute, e.g., entirely on processor 3086, partly on processor 3086 andpartly on a remote computer connected to network 3050, or entirely onthe remote computer.

Illustrative Embodiments And Clauses

Any of these embodiments or clauses can be carried out bydata-processing system 3000, as described herein with reference to FIG.30. The data-processing system can permit interpreting sensor data ofthe listed user gestures or actions and providing the listed effects on,e.g., virtual 3D models.

Embodiment 1 (e.g., MobiSpin, above): Use a mobile device to detect areference plane and a virtual pointer in a 3D interaction space definedinside a separate virtual space such that the reference plane and thepointer can be:

-   -   Rotated directly by rotating the mobile device;    -   Translated by using touch gestures by the user on the screen of        the mobile device OR by directly detecting the location of the        mobile device using a 3D camera system.    -   Used for selecting 3D virtual object, in whole or in parts    -   Used for changing the view of the scene rendered to visually        describe the separate virtual space either on a separate display        screen, a virtual reality device, an augmented reality device,        or a completely different computer.

Embodiment 2: Embodiment 1 such that the reference plane (FloatingCanvas) associated with mobile device can be:

-   -   Used to select a single object from a collection of multiple        objects in the separate virtual space    -   Used to select a region on the surface of a 3D object in the        separate virtual space    -   Used to translate and rotate a virtual object after selecting        the object    -   Used to create planar and non-planar curves in the separate        virtual space    -   Used to change the viewing direction and location of the scene        describing the virtual space by translating and rotating a        virtual camera that is being used to render the scene

Embodiment 3: Embodiment 1 or 2 such that the pointing line (LaserPointer) associated with mobile device can be:

-   -   Used to select a single object from a collection of multiple        objects in the separate virtual space    -   Used to select a region on the surface of a 3D object in the        separate virtual space    -   Used to translate and rotate a virtual object after selecting        the object    -   Used to change the viewing direction and location of the scene        describing the virtual space by translating and rotating a        virtual camera that is being used to render the scene

Embodiment 4: (e.g., MobiCAD): Use of a mobile device to detect a datumplane that is used to create, select, modify and assemble whole or partsof virtual objects wherein:

-   -   The category of virtual objects includes a parametric and        feature-based 3D models (including straight protrusions and        solids of revolution) defined by the boundary representation of        the volume enclosed by the object;    -   The creation of the protrusions is performed by:        -   Aligning the mobile device to one of the standard reference            planes (top, side, and front) incremental and dynamic            addition of cross-sections        -   Subsequently, sketching the shape of the object as seen in            the aligned view (plan, elevation)        -   Finally, specifying the magnitude of protrusion by using the            mobile device screen in input numerical values.    -   The creation of the solids of revolution is performed by:        -   Aligning the mobile device to one of the standard reference            planes (top, side, and front)        -   Subsequently, sketching the shape of the object as seen in            the aligned view (plan, elevation)        -   Finally, specifying the angle of revolution by using the            mobile device screen in input numerical values.    -   The orientation of the mobile device is used to direct a virtual        pointer in the virtual space to select distinct features such as        vertices, edges, and faces on the 3D objects    -   The modification of the protrusions and solids of revolution is        achieved by:        -   Selecting a distinct feature (e.g. the sketch of the            protrusion) by aligning the mobile device to one of the            standard reference planes (top, side, and front)        -   Subsequently, sketching the shape of the object as seen in            the aligned view (plan, elevation) or specifying the            dimension of revolution by using the mobile device screen in            input numerical values.    -   The assembly of multiple parts into a single meaningful        representation of a designed object is achieved by:        -   Searching for parts from a repository using mobile menus        -   Configuring each part with respect to each other by using            the combination mobile tilt and touch gestures.

Embodiment 5 (e.g., MobiSweep): Use of a mobile device to detect areference plane in a 3D interaction space defined inside a separatevirtual space to create, modify, deform, and manipulate virtual objectswherein:

-   -   The category of virtual objects includes a generalized cylinder        defined by sections along a skeletal axis;    -   The creation of the generalized cylinder is performed by        incremental and dynamic addition of cross-sections using one        finger and three finger gestures applied on the mobile device        screen.    -   The modification of the generalized cylinder include:        -   Translating the reference plane so as to intersect a desired            cylinder        -   Use of finger gesture on the mobile device screen to specify            the intent to select the intersected cylinder.        -   Use of two finger gestures to scale the cylinder    -   The deformation of the generalized cylinder include:        -   Dynamically bending about the skeletal axis by rotating the            top-most section of the cylinder using the rotation of the            mobile device,        -   Changing the shape of the top-most section of the cylinder            by providing a sketch of the new section through the screen            of the a section        -   Scaling of a section using two finger gestures

-   Deformation of a section.    -   The manipulation of the generalized cylinder is achieved by:        -   Translating the reference plane so as to intersect a desired            cylinder        -   Use of finger gesture on the mobile device screen to specify            the intent to select the intersected cylinder        -   Use of two finger gestures to move the cylinder        -   Use of the mobile device's orientation to rotate the            cylinder

Embodiment 6 (e.g., MobiForm): Use of a mobile device to detect areference plane for drawing on, texturing, and deforming arbitrarypolygon meshes to compose 3D digital designs in a 3D virtual spacedisplayed on a separate computing device with a display screen. Thedeformation is achieved by:

-   -   Rotating the smartphone so as to move the reference plane on the        surface of the virtual 3D objects represented as a polygon mesh    -   Use of one finger gesture to create inward deformation, sketched        entities, and texturing    -   Use of two finger gestures to create outward deformation on the        surface of the 3D virtual objects    -   Use of additional buttons (e.g. volume button) to switch from        the modeling mode to the rotation mode and vice-versa.

Embodiment 7 (e.g., MobiSpace): Use of a mobile device as a mid-air 3Dcontroller, without including additional hardware on the device.

-   -   Combining the tilt sensor data from a mobile device with 3D        depth data acquired from an externally mounted depth sensor to        track hand-held motion of the device in 3D space.    -   Augmenting mid-air tracking of a mobile device with concurrent        multi-touch gestures on its surface to support both spatial and        discrete inputs during mid-air interactions with a 3D modeling        system.        -   Clutching and spatially manipulating virtual 3D objects.        -   Deforming a virtual 3D object by clutching a portion its            geometry, and moving the device in the direction of            elongation, compression, bending, or twisting.        -   Uniform or constrained scaling of virtual 3D objects.        -   Applying context-based constraints to interpret coarse            mid-air inputs into more precise and refined modeling            operations.        -   Beautifying assembly structure through geometric and spatial            constraints between adjoining shapes.        -   Patterning a shape along a linear direction or the outline            of an adjoining shape.    -   Configuring and assembling multiple objects in 3D space to        compose a virtual design or art prototype.        -   Assembling planar shapes drawn on the mobile device GUI.        -   Assembling 3D shapes acquired from digital 3D scanning or a            3D model repository. (similar to before but doing it now by            moving the mobile device)

Clause 1.A method for manipulating a 3D virtual object displayed in avirtual space using a mobile device having a touch screen, the methodcomprising: defining a reference plane in the virtual space; changing alocation of the reference plane in the virtual space based at least inpart on an input received by the mobile device; selecting all or part ofthe 3D virtual object that is intersected by the reference plane;receiving an indication of a manipulation performed on the mobiledevice; and modifying a representation of all or part of the 3D virtualobject intersected by the reference plane based at least in part on themanipulation performed on the mobile device.

Clause 2. The method of Clause 1, further comprising rendering thevirtual space on a 2D display device, a virtual reality device, or anaugmented reality device.

Clause 3. The method of any of Clauses 1-2, wherein the manipulationcomprises a rotation of the mobile device and the modifying therepresentation comprises rotating all or part of the 3D virtual objectthat is intersected by the reference plane.

Clause 4. The method of any of Clauses 1-3, wherein the manipulationcomprises a pinch gesture on the touch screen of the mobile device andmodifying the representation comprises shrinking all or part of the 3Dvirtual object that is intersected by the reference plane.

Clause 5. The method of any of Clauses 1-4, wherein the manipulationcomprises a curve drawn on the touch screen of the mobile device andmodifying the representation comprises creating a planar curve in thevirtual space.

Clause 6. The method of any of Clauses 1-5, wherein the manipulationcomprises a multi-touch drag gesture across the touch screen of themobile device and modifying the representation comprises translating allor part of the 3D virtual object that is intersected by the referenceplane.

Clause 7. A method for manipulating a 3D virtual object displayed in avirtual space a using a mobile device having a touch screen, the methodcomprising: positioning a pointer in the virtual space based at least inpart on an orientation of a plane defined by the touch screen of themobile device; correlating all or part of the 3D virtual object with alocation of the pointer in the virtual space; receiving an indication ofa selection command performed on the mobile phone; selecting at least aportion of the 3D virtual object correlated with the location of thepointer in response to the received indication; and receiving anindication of movement of the mobile device; and manipulating theportion of the 3D virtual object based at least in part on theindication of movement.

Clause 8. The method of Clause 7, wherein the manipulation comprises atleast one of translating the portion of the 3D virtual object, rotatingthe portion of the 3D virtual object, or changing a viewing direction ofat least the portion of the 3D virtual object.

Clause 9. A method for manipulating a 3D virtual object displayed in avirtual space a using a mobile device, the method comprising: receivingan indication of an orientation of the mobile device, the orientation ofthe mobile device detected by at least one of an accelerometer orgyroscope in the mobile device; receiving an indication of a position ofthe mobile device, the position detected by a position sensor configuredto detect the position of the mobile device in space relative to theposition sensor; manipulating the 3D virtual object in the virtual spacebased on both the orientation of the mobile device and the position ofthe mobile device.

Clause 10. The method of Clause 9, wherein manipulating the 3D virtualobject comprises modifying geometric and spatial constraints withrespect to a different 3D virtual object.

Clause 11. The method of any of Clauses 9-10, wherein manipulating the3D virtual object comprises creating a new 3D virtual object, thecreating based at least in part on a change in the position of themobile device along a linear direction.

Clause 12. The method of Clause 11, wherein the creating a new 3Dvirtual object comprises creating at least one of a sweep creating acurve across multiple frames, an assembly of multiple existing 3Dvirtual objects, a pattern repeated shapes, or a deformation of a shape.

Clause 13. The method of any of Clauses 9-12, further comprisingreceiving an indication of an input received on a touch screen of themobile device.

Clause 14. The method of any of Clauses 9-13, wherein the positionsensor detects the position of the mobile device by at least one ofmachine vision, ultrasound, or magnetic sensing.

Clause 15. A computing apparatus, the computing apparatus comprising: aprocessor; and a memory storing instructions that, when executed by theprocessor, configure the apparatus to perform a method as described inany of Clauses 1-14.

Clause 16. A non-transitory computer-readable storage medium formanipulating a 3D virtual object displayed in a virtual space a using amobile device, the computer-readable storage medium includinginstructions that when executed by a computer, cause the computer toperform a method as described in any of Clauses 1-14.

Clause 17. A method for manipulating a threeD virtual object displayedin a virtual space using a mobile device having a touch screen, themethod comprising: receiving an input at the mobile device; changing alocation of a reference plane in the virtual space based at least inpart on the input; selecting all or part of the threeD virtual objectthat is intersected by the reference plane; receiving an indication of amanipulation performed on the mobile device; and modifying arepresentation of all or part of the threeD virtual object intersectedby the reference plane based at least in part on the indication of themanipulation performed on the mobile device.

Clause 18. The method of Clause 17, further comprising rendering thevirtual space on a 2D display device, a virtual reality device, or anaugmented reality device.

Clause 19. The method of Clause 17 or 18, wherein the manipulationcomprises a rotation of the mobile device and the modifying therepresentation comprises rotating all or part of the threeD virtualobject that is intersected by the reference plane.

Clause 20. The method of any of Clauses 17-19, wherein the manipulationcomprises a pinch gesture on the touch screen of the mobile device andmodifying the representation comprises shrinking all or part of thethreeD virtual object that is intersected by the reference plane.

Clause 21. The method of any of Clauses 17-20, wherein the manipulationcomprises a curve drawn on the touch screen of the mobile device andmodifying the representation comprises creating a planar curve in thevirtual space.

Clause 22. The method of any of Clauses 17-21, wherein the manipulationcomprises a multi-touch drag gesture across the touch screen of themobile device and modifying the representation comprises translating allor part of the threeD virtual object that is intersected by thereference plane.

Clause 23. A method for manipulating a threeD virtual object displayedin a virtual space using a mobile device having a touch screen, themethod comprising: receiving an indication of an orientation of a planedefined by the touch screen of the mobile device; positioning a pointerat a location in the virtual space based at least in part on theindication of the orientation; correlating at least part of the threeDvirtual object with the location of the pointer in the virtual space;receiving an indication of a selection command performed on the mobiledevice; selecting at least a portion of the threeD virtual objectcorrelated with the location of the pointer in response to the receivedindication of the selection command; receiving an indication of movementof the mobile device; and manipulating the portion of the threeD virtualobject based at least in part on the indication of movement.

Clause 24. The method of Clause 23, wherein the manipulating the portionof the threeD virtual object comprises at least one of: translating theportion of the threeD virtual object, rotating the portion of the threeDvirtual object, or changing a viewing direction of at least the portionof the threeD virtual object.

Clause 25. A method for manipulating a first threeD virtual objectdisplayed in a virtual space using a mobile device, the methodcomprising: receiving an indication of an orientation of the mobiledevice, the orientation of the mobile device detected by at least one ofan accelerometer or gyroscope in the mobile device; receiving anindication of a position of the mobile device, the position detected bya position sensor configured to detect the position of the mobile devicein space relative to the position sensor; and manipulating the firstthreeD virtual object in the virtual space based on both the orientationof the mobile device and the position of the mobile device.

Clause 26. The method of Clause 25, wherein manipulating the threeDvirtual object comprises modifying geometric and spatial constraintswith respect to a second, different threeD virtual object.

Clause 27. The method of Clause 25 or 26, wherein manipulating the firstthreeD virtual object comprises creating a third, new threeD virtualobject based at least in part on a change in the position of the mobiledevice along a linear direction.

Clause 28. The method of Clause 27, wherein the creating the third, newthreeD virtual object comprises creating at least one of: a curve acrossmultiple frames, an assembly of multiple existing threeD virtualobjects, a pattern of repeated shapes, or a deformed shape.

Clause 29. The method of any of Clauses 25-28, further comprisingreceiving an indication of an input received on the mobile device,wherein the input comprises a rotation of the mobile device and themanipulating the first threeD virtual object comprises rotating all orpart of the first threeD virtual object is the virtual space.

Clause 30. The method of any of Clauses 25-29, further comprisingreceiving an indication of an input received on a touch screen of themobile device, wherein the input comprises a pinch gesture on the touchscreen of the mobile device and the manipulating the first threeDvirtual object comprises shrinking all or part of the first threeDvirtual object.

Clause 31. The method of any of Clauses 25-30, further comprisingreceiving an indication of an input received on a touch screen of themobile device, wherein the input comprises a curve drawn on the touchscreen of the mobile device and the manipulating the first threeDvirtual object comprises creating a planar curve is the virtual space.

Clause 32. The method of any of Clauses 25-31, further comprisingreceiving an indication of an input received on a touch screen of themobile device, wherein the input comprises multi-touch drag gestureacross the touch screen of the mobile device and the manipulating thefirst threeD virtual object comprises translating all or part of thefirst threeD virtual object is the virtual space.

Clause 33. The method of any of Clauses 25-32, wherein the positionsensor detects the position of the mobile device by at least one ofmachine vision, ultrasound, or magnetic sensing.

Clause 34. A computing apparatus, the computing apparatus comprising: aprocessor; and a memory storing instructions that, when executed by theprocessor, configure the apparatus to perform a method as described inany of Clauses 17-33.

Clause 35. A non-transitory computer-readable storage medium formanipulating a 3D virtual object displayed in a virtual space a using amobile device, the computer-readable storage medium includinginstructions that when executed by a computer, cause the computer toperform a method as described in any of Clauses 17-33.

CONCLUSION

The invention is inclusive of combinations of the aspects describedherein. References to “a particular aspect” (or “embodiment” or“version”) and the like refer to features that are present in at leastone aspect of the invention. Separate references to “an aspect” (or“embodiment”) or “particular aspects” or the like do not necessarilyrefer to the same aspect or aspects; however, such aspects are notmutually exclusive, unless so indicated or as are readily apparent toone of skill in the art. The use of singular or plural in referring to“method” or “methods” and the like is not limiting. The word “or” isused in some examples in a non-exclusive sense, unless otherwiseexplicitly noted.

The invention has been described in detail with particular reference tocertain preferred aspects thereof, but it will be understood thatvariations, combinations, and modifications can be effected by a personof ordinary skill in the art within the spirit and scope of theinvention.

1-15. (canceled)
 16. A method for manipulating a 3D virtual objectdisplayed in a virtual space using a mobile device having a touchscreen, the method comprising: receiving an input at the mobile device;changing a location of a reference plane in the virtual space based atleast in part on the input; selecting all or part of the 3D virtualobject that is intersected by the reference plane; receiving anindication of a manipulation performed on the mobile device; andmodifying a representation of all or part of the 3D virtual objectintersected by the reference plane based at least in part on theindication of the manipulation performed on the mobile device.
 17. Themethod of claim 16, further comprising rendering the virtual space on a2D display device, a virtual reality device, or an augmented realitydevice.
 18. The method of claim 16, wherein the manipulation comprises arotation of the mobile device and the modifying the representationcomprises rotating all or part of the 3D virtual object that isintersected by the reference plane.
 19. The method of claim 16, whereinthe manipulation comprises a pinch gesture on the touch screen of themobile device and modifying the representation comprises shrinking allor part of the 3D virtual object that is intersected by the referenceplane.
 20. The method of claim 16, wherein the manipulation comprises acurve drawn on the touch screen of the mobile device and modifying therepresentation comprises creating a planar curve in the virtual space.21. The method of claim 16, wherein the manipulation comprises amulti-touch drag gesture across the touch screen of the mobile deviceand modifying the representation comprises translating all or part ofthe 3D virtual object that is intersected by the reference plane.
 22. Amethod for manipulating a 3D virtual object displayed in a virtual spaceusing a mobile device having a touch screen, the method comprising:receiving an indication of an orientation of a plane defined by thetouch screen of the mobile device; positioning a pointer at a locationin the virtual space based at least in part on the indication of theorientation; correlating at least part of the 3D virtual object with thelocation of the pointer in the virtual space; receiving an indication ofa selection command performed on the mobile device; selecting at least aportion of the 3D virtual object correlated with the location of thepointer in response to the received indication of the selection command;receiving an indication of movement of the mobile device; andmanipulating the portion of the 3D virtual object based at least in parton the indication of movement.
 23. The method of claim 22, wherein themanipulating the portion of the 3D virtual object comprises at least oneof: translating the portion of the 3D virtual object, rotating theportion of the 3D virtual object, or changing a viewing direction of atleast the portion of the 3D virtual object.
 24. A method formanipulating a first 3D virtual object displayed in a virtual spaceusing a mobile device, the method comprising: receiving an indication ofan orientation of the mobile device, the orientation of the mobiledevice detected by at least one of an accelerometer or gyroscope in themobile device; receiving an indication of a position of the mobiledevice, the position detected by a position sensor configured to detectthe position of the mobile device in space relative to the positionsensor; and manipulating the first 3D virtual object in the virtualspace based on both the orientation of the mobile device and theposition of the mobile device.
 25. The method of claim 24, whereinmanipulating the 3D virtual object comprises modifying geometric andspatial constraints with respect to a second, different 3D virtualobject.
 26. The method of claim 24, wherein manipulating the first 3Dvirtual object comprises creating a third, new 3D virtual object basedat least in part on a change in the position of the mobile device alonga linear direction by creating at least one of: a curve across multipleframes, an assembly of multiple existing 3D virtual objects, a patternof repeated shapes, or a deformed shape.
 27. The method of claim 24,further comprising receiving an indication of an input received on themobile device, wherein the input comprises a rotation of the mobiledevice and the manipulating the first 3D virtual object comprisesrotating all or part of the first 3D virtual object is the virtualspace.
 28. The method of claim 24, further comprising receiving anindication of an input received on a touch screen of the mobile device,wherein the input comprises a pinch gesture on the touch screen of themobile device and the manipulating the first 3D virtual object comprisesshrinking all or part of the first 3D virtual object.
 29. The method ofclaim 24, further comprising receiving an indication of an inputreceived on a touch screen of the mobile device, wherein the inputcomprises a curve drawn on the touch screen of the mobile device and themanipulating the first 3D virtual object comprises creating a planarcurve is the virtual space.
 30. The method of claim 24, furthercomprising receiving an indication of an input received on a touchscreen of the mobile device, wherein the input comprises multi-touchdrag gesture across the touch screen of the mobile device and themanipulating the first 3D virtual object comprises translating all orpart of the first 3D virtual object is the virtual space.